[common] Introduce BitmapFileIndexMetaV2. #5028
Conversation
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java
Show resolved
Hide resolved
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java
Show resolved
Hide resolved
|
+1 |
|
I found another question that needs to be discussed. Should the index version default to v1 or v2? @Tan-JiaLiang @JingsongLi |
I don't have a clear idea, although it may cause compatibility issues, bitmap should not be used too much. Maintaining the default version to 1 is a relatively conservative choice. |
paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java
Show resolved
Hide resolved
| * ｜ ... ｜ | ||
| * +-------------------------------------------------+----------------- | ||
| * | ||
| * index block format: |
There was a problem hiding this comment.
Maybe we should have ability to compress index block?
| Options options; | ||
|
|
||
| void tryDeserialize() { | ||
| if (entryList == null) { |
There was a problem hiding this comment.
Maybe you can take a look to https://cwiki.apache.org/confluence/display/PAIMON/PIP-25%3A+Introduce+a+key-value+file+format+for+paimon+primary+key+table
Here just something like LevelDB sst block, we can store length to avoid deserialize all entries.
Purpose
When the bitmap-indexed column cardinality is high, using the first version of the bitmap index format will take a lot of time to read the entire dictionary. But in fact we don't need a full dictionary when dealing with a small number of predicates, the performance of predicate hits on the bitmap can be improved by creating a secondary index on the dictionary.
https://docs.google.com/document/d/11dJlGlSX3dwYKKrPN0DQ2XQTsx6d9wI6DTBIiiBwomM/edit?tab=t.0
performance
cardinality 1000:
cardinality 10000:
cardinality 30000:
cardinality 50000:
cardinality 80000:
cardinality 100000:
Tests
org.apache.paimon.fileindex.bitmapindex.TestBitmapFileIndex
org.apache.paimon.spark.SparkFileIndexITCase
org.apache.paimon.benchmark.bitmap.BitmapIndexBenchmark
API and Format
Documentation
docs/content/concepts/spec/fileindex.md